Considering Subjects and Scenarios in Large-Scale User-Centered Evaluation of a Multilingual Multimodal Medical Search System

Khresmoi aims to provide a multilingual and multimodal search and access system for biomedical information and documents, which targets three classes of user. Two of these are groups with general medical interests: the general public and general medical practitioners. The other is an example clinician group with a specific expertise: radiologists. Khresmoi targets diverse users, with: varying native language, language skills, medical knowledge, information needs or querying behaviour. The system seeks to provide these groups with innovative and effective services for searching through the large amount of available medical information. It gathers several innovative technologies linked to medical information (e.g. information extraction and retrieval, machine translation) to provide a comprehensive tool adapted to all users. In parallel to the development of the search system, a global evaluation strategy is being designed to enable an assessment of the efficiency and effectiveness of the developed technologies and of the quality of the support provided to the target users. These evaluation plans consist of both empirical and user-centred evaluations. We focus here on user-centred evaluations. Creation of holistic user-centred evaluation approaches allows to evaluate the search application in a comprehensive way while being mindful of the diversity of users. First steps in this strategy relate to the experimental subjects and evaluation scenarios. Firstly, demographic features of each class of user have to be defined in order to get representative groups. Gaining subjects of this nature can be problematic, especially when medical professionals are required, and it raises issues such as the payment of subjects, or the geographic distance between subjects and investigators. Moreover, the Khresmoi system is intended for use by individuals with different native languages and different skills in the English language. Again, this raises issues such as gaining a good spread of subjects with different language skills, as well as the assessment of subjects' language skills. To determine the demographic spread of subjects, standard questionnaires should be completed by subjects to determine their age, gender, medical condition, prior use of medical search engines, computer skill, etc. Secondly, a robust methodology for scenario creation has to be developed. User studies conducted within the project gave us an insight into the types of search tasks that users are performing and would perform in the medical space and a method to model this task information will be defined. Results of these evaluations will guide further research and development in the Khresmoi project.